ebab7d0da53c51438720d60c6e2e467c - /page/wikibook/entry.mu

Micron Document

<!DOCTYPE html>

<title>Search engine indexing</title>

</head>

<span id="openzim-page-title" class="mw-page-title-main"><span class="mw-page-title-main">Search engine indexing</span></span>

</h1>

</header>

<p><b>Search engine indexing</b> is the collecting, <a href="Parsing" title="Parsing">parsing</a>, and storing of data to facilitate fast and accurate <a href="Information_retrieval" title="Information retrieval">information retrieval</a>. Index design incorporates interdisciplinary concepts from <a href="Linguistics" title="Linguistics">linguistics</a>, <a href="Cognitive_psychology" title="Cognitive psychology">cognitive psychology</a>, mathematics, <a href="Informatics" title="Informatics">informatics</a>, and <a href="Computer_science" title="Computer science">computer science</a>. An alternate name for the process, in the context of <a href="Search_engine" title="Search engine">search engines</a> designed to find <a href="Web_page" title="Web page">web pages</a> on the Internet, is <i><a href="Web_indexing" title="Web indexing">web indexing</a></i>.

</p><p>Popular search engines focus on the <a href="Full-text_search" title="Full-text search">full-text</a> indexing of online, <a href="Natural_language_processing" title="Natural language processing">natural language</a> documents.<sup id="cite_ref-1" class="reference"><a href="#cite_note-1"><span class="cite-bracket">[</span>1<span class="cite-bracket">]</span></a></sup> <a href="Media_type" title="Media type">Media types</a> such as pictures, video, audio,<sup id="cite_ref-2" class="reference"><a href="#cite_note-2"><span class="cite-bracket">[</span>2<span class="cite-bracket">]</span></a></sup> and graphics<sup id="cite_ref-3" class="reference"><a href="#cite_note-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup> are also searchable.

</p><p><a href="Metasearch_engine" title="Metasearch engine">Meta search engines</a> reuse the indices of other services and do not store a local index whereas cache-based search engines permanently store the index along with the <a href="Text_corpus" title="Text corpus">corpus</a>. Unlike full-text indices, partial-text services restrict the depth indexed to reduce index size. Larger services typically perform indexing at a predetermined time interval due to the required time and processing costs, while <a href="Intelligent_agent" title="Intelligent agent">agent</a>-based search engines index in <a href="Real_time_business_intelligence" class="mw-redirect" title="Real time business intelligence">real time</a>.

</p>

<div class="mw-heading mw-heading2"><h2 id="Indexing">Indexing</h2></div>

<p>The purpose of storing an index is to optimize speed and performance in finding <a href="Relevance_(information_retrieval)" title="Relevance (information retrieval)">relevant</a> documents for a search query. Without an index, the search engine would <a href="Lexical_analysis" title="Lexical analysis">scan</a> every document in the <a href="Text_corpus" title="Text corpus">corpus</a>, which would require considerable time and computing power. For example, while an index of 10,000 documents can be queried within milliseconds, a sequential scan of every word in 10,000 large documents could take hours. The additional <a href="Computer_Storage" class="mw-redirect" title="Computer Storage">computer storage</a> required to store the index, as well as the considerable increase in the time required for an update to take place, are traded off for the time saved during information retrieval.

</p>

<div class="mw-heading mw-heading3"><h3 id="Index_design_factors">Index design factors</h3></div>

<p>Major factors in designing a search engine's architecture include:

</p>

<dl><dt>Merge factors</dt>

<dd>How data enters the index, or how words or subject features are added to the index during text corpus traversal, and whether multiple indexers can work asynchronously. The indexer must first check whether it is updating old content or adding new content. Traversal typically correlates to the <a href="Web_crawling" class="mw-redirect" title="Web crawling">data collection</a> policy. Search engine index merging is similar in concept to the <a href="Merge_(SQL)" title="Merge (SQL)">SQL Merge</a> command and other merge algorithms.<sup id="cite_ref-4" class="reference"><a href="#cite_note-4"><span class="cite-bracket">[</span>4<span class="cite-bracket">]</span></a></sup></dd>

<dt>Storage techniques</dt>

<dd>How to store the index <a href="Data" title="Data">data</a>, that is, whether information should be data compressed or filtered.</dd>

<dt>Index size</dt>

<dd>How much <a href="Computer_data_storage" title="Computer data storage">computer storage</a> is required to support the index.</dd>

<dt>Lookup speed</dt>

<dd>How quickly a word can be found in the <a href="Inverted_index" title="Inverted index">inverted index</a>. The speed of finding an entry in a data structure, compared with how quickly it can be updated or removed, is a central focus of computer science.</dd>

<dt>Maintenance</dt>

<dd>How the index is maintained over time.<sup id="cite_ref-5" class="reference"><a href="#cite_note-5"><span class="cite-bracket">[</span>5<span class="cite-bracket">]</span></a></sup></dd>

<dt>Fault tolerance</dt>

<dd>How important it is for the service to be reliable. Issues include dealing with index corruption, determining whether bad data can be treated in isolation, dealing with bad hardware, <a href="Partition_(database)" title="Partition (database)">partitioning</a>, and schemes such as <a href="Hash_function" title="Hash function">hash-based</a> or composite partitioning,<sup id="cite_ref-6" class="reference"><a href="#cite_note-6"><span class="cite-bracket">[</span>6<span class="cite-bracket">]</span></a></sup> as well as <a href="Replication_(computer_science)" class="mw-redirect" title="Replication (computer science)">replication</a>.</dd></dl>

<div class="mw-heading mw-heading3"><h3 id="Index_data_structures">Index data structures</h3></div>

<p>Search engine architectures vary in the way indexing is performed and in methods of index storage to meet the various design factors.

</p>

<dl><dt><a href="Suffix_tree" title="Suffix tree">Suffix tree</a></dt>

<dd>Figuratively structured like a tree, supports linear time lookup. Built by storing the suffixes of words. The suffix tree is a type of <a href="Trie" title="Trie">trie</a>. Tries support <a href="Extendible_hashing" title="Extendible hashing">extendible hashing</a>, which is important for search engine indexing.<sup id="cite_ref-7" class="reference"><a href="#cite_note-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup> Used for searching for patterns in <a href="DNA" title="DNA">DNA</a> sequences and clustering. A major drawback is that storing a word in the tree may require space beyond that required to store the word itself.<sup id="cite_ref-Gus97_8-0" class="reference"><a href="#cite_note-Gus97-8"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup> An alternate representation is a <a href="Suffix_array" title="Suffix array">suffix array</a>, which is considered to require less virtual memory and supports data compression such as the <a href="Burrows%E2%80%93Wheeler_transform" title="Burrows–Wheeler transform">BWT</a> algorithm.</dd></dl>

<dl><dt><a href="Inverted_index" title="Inverted index">Inverted index</a></dt>

<dd>Stores a list of occurrences of each atomic search criterion,<sup id="cite_ref-9" class="reference"><a href="#cite_note-9"><span class="cite-bracket">[</span>9<span class="cite-bracket">]</span></a></sup> typically in the form of a <a href="Hash_table" title="Hash table">hash table</a> or <a href="Binary_tree" title="Binary tree">binary tree</a>.<sup id="cite_ref-10" class="reference"><a href="#cite_note-10"><span class="cite-bracket">[</span>10<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-11" class="reference"><a href="#cite_note-11"><span class="cite-bracket">[</span>11<span class="cite-bracket">]</span></a></sup></dd></dl>

<dl><dt><a href="Citation_index" title="Citation index">Citation index</a></dt>

<dd>Stores citations or hyperlinks between documents to support citation analysis, a subject of <a href="Bibliometrics" title="Bibliometrics">bibliometrics</a>.</dd>

<dt><a href="N-gram" title="N-gram"><i>n</i>-gram index</a></dt>

<dd>Stores sequences of length of data to support other types of retrieval or <a href="Text_mining" title="Text mining">text mining</a>.<sup id="cite_ref-12" class="reference"><a href="#cite_note-12"><span class="cite-bracket">[</span>12<span class="cite-bracket">]</span></a></sup></dd>

<dt><a href="Document-term_matrix" title="Document-term matrix">Document-term matrix</a></dt>

<dd>Used in latent semantic analysis, stores the occurrences of words in documents in a two-dimensional <a href="Sparse_matrix" title="Sparse matrix">sparse matrix</a>.</dd></dl>

<div class="mw-heading mw-heading3"><h3 id="Challenges_in_parallelism">Challenges in parallelism</h3></div>

<p>A major challenge in the design of search engines is the management of serial computing processes. There are many opportunities for <a href="Race_conditions" class="mw-redirect" title="Race conditions">race conditions</a> and coherent faults. For example, a new document is added to the corpus and the index must be updated, but the index simultaneously needs to continue responding to search queries. This is a collision between two competing tasks. Consider that authors are producers of information, and a <a href="Web_crawler" title="Web crawler">web crawler</a> is the consumer of this information, grabbing the text and storing it in a cache (or <a href="Text_corpus" title="Text corpus">corpus</a>). The forward index is the consumer of the information produced by the corpus, and the inverted index is the consumer of information produced by the forward index. This is commonly referred to as a <b>producer-consumer model</b>. The indexer is the producer of searchable information and users are the consumers that need to search. The challenge is magnified when working with distributed storage and distributed processing. In an effort to scale with larger amounts of indexed information, the search engine's architecture may involve <a href="Distributed_computing" title="Distributed computing">distributed computing</a>, where the search engine consists of several machines operating in unison. This increases the possibilities for incoherency and makes it more difficult to maintain a fully synchronized, distributed, parallel architecture.<sup id="cite_ref-13" class="reference"><a href="#cite_note-13"><span class="cite-bracket">[</span>13<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Inverted_indices">Inverted indices</h3></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Inverted_index" title="Inverted index">Inverted index</a></div>

<p>Many search engines incorporate an <a href="Inverted_index" title="Inverted index">inverted index</a> when evaluating a <a href="Web_search_query" class="mw-redirect" title="Web search query">search query</a> to quickly locate documents containing the words in a query and then rank these documents by relevance. Because the inverted index stores a list of the documents containing each word, the search engine can use direct <a href="Random_access" title="Random access">access</a> to find the documents associated with each word in the query in order to retrieve the matching documents quickly. The following is a simplified illustration of an inverted index:

</p>

<caption>Inverted index

</caption>

<th>Documents

</th></tr>

<tr>

<td>Document 1, Document 3, Document 4, Document 5, Document 7

</td></tr>

<tr>

<td>Document 2, Document 3, Document 4

</td></tr>

<tr>

<td>Document 5

</td></tr>

<tr>

<td>Document 7

</td></tr></tbody></table>

<p>This index can only determine whether a word exists within a particular document, since it stores no information regarding the frequency and position of the word; it is therefore considered to be a <a href="Boolean_data_type" title="Boolean data type">Boolean</a> index. Such an index determines which documents match a query but does not rank matched documents. In some designs the index includes additional information such as the frequency of each word in each document or the positions of a word in each document.<sup id="cite_ref-14" class="reference"><a href="#cite_note-14"><span class="cite-bracket">[</span>14<span class="cite-bracket">]</span></a></sup> Position information enables the search algorithm to identify word proximity to support searching for phrases; frequency can be used to help in ranking the relevance of documents to the query. Such topics are the central research focus of <a href="Information_retrieval" title="Information retrieval">information retrieval</a>.

</p><p>The inverted index is a <a href="Sparse_matrix" title="Sparse matrix">sparse matrix</a>, since not all words are present in each document. To reduce <a href="Computer_Storage" class="mw-redirect" title="Computer Storage">computer storage</a> memory requirements, it is stored differently from a two dimensional <a href="Array_data_structure" class="mw-redirect" title="Array data structure">array</a>. The index is similar to the <a href="Document-term_matrix" title="Document-term matrix">term document matrices</a> employed by <a href="Latent_semantic_analysis" title="Latent semantic analysis">latent semantic analysis</a>. The inverted index can be considered a form of a hash table. In some cases the index is a form of a <a href="Binary_tree" title="Binary tree">binary tree</a>, which requires additional storage but may reduce the lookup time. In larger indices the architecture is typically a <a href="Distributed_hash_table" title="Distributed hash table">distributed hash table</a>.<sup id="cite_ref-15" class="reference"><a href="#cite_note-15"><span class="cite-bracket">[</span>15<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading4"><h4 id="Implementation_of_Phrase_Search_Using_an_Inverted_Index">Implementation of Phrase Search Using an Inverted Index</h4></div>

<p>For phrase searching, a specialized form of an inverted index called a positional index is used. A positional index not only stores the ID of the document containing the token but also the exact position(s) of the token within the document in the <a href="Postings_list" title="Postings list">postings list</a>. The occurrences of the phrase specified in the query are retrieved by navigating these postings list and identifying the indexes at which the desired terms occur in the expected order (the same as the order in the phrase). So if we are searching for occurrence of the phrase "First Witch", we would:

</p>

<ol><li>Retrieve the postings list for "first" and "witch"</li>

<li>Identify the first time that "witch" occurs after "first"</li>

<li>Check that this occurrence is immediately after the occurrence of "first".</li>

<li>If not, continue to the next occurrence of "first".</li></ol>

<p>The postings lists can be navigated using a binary search in order to minimize the time complexity of this procedure.<sup id="cite_ref-16" class="reference"><a href="#cite_note-16"><span class="cite-bracket">[</span>16<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Index_merging">Index merging</h3></div>

<p>The inverted index is filled via a merge or rebuild. A rebuild is similar to a merge but first deletes the contents of the inverted index. The architecture may be designed to support incremental indexing,<sup id="cite_ref-17" class="reference"><a href="#cite_note-17"><span class="cite-bracket">[</span>17<span class="cite-bracket">]</span></a></sup> where a merge identifies the document or documents to be added or updated and then parses each document into words. For technical accuracy, a merge conflates newly indexed documents, typically residing in virtual memory, with the index cache residing on one or more computer hard drives.

</p><p>After parsing, the indexer adds the referenced document to the document list for the appropriate words. In a larger search engine, the process of finding each word in the inverted index (in order to report that it occurred within a document) may be too time consuming, and so this process is commonly split up into two parts, the development of a forward index and a process which sorts the contents of the forward index into the inverted index. The inverted index is so named because it is an inversion of the forward index.

</p>

<div class="mw-heading mw-heading3"><h3 id="The_forward_index">The forward index</h3></div>

<p>The forward index stores a list of words for each document. The following is a simplified form of the forward index:

</p>

<caption>Forward Index

</caption>

<th>Document</th>

<th>Words

</th></tr>

<tr>

<td>Document 1</td>

<td>the,cow,says,moo

</td></tr>

<tr>

<td>Document 2</td>

<td>the,cat,and,the,hat

</td></tr>

<tr>

<td>Document 3</td>

<td>the,dish,ran,away,with,the,spoon

</td></tr></tbody></table>

<p>The rationale behind developing a forward index is that as documents are parsed, it is better to intermediately store the words per document. The delineation enables asynchronous system processing, which partially circumvents the inverted index update <a href="https://en.wiktionary.org/wiki/bottleneck" class="extiw external" title="wikt:bottleneck">bottleneck</a>.<sup id="cite_ref-18" class="reference"><a href="#cite_note-18"><span class="cite-bracket">[</span>18<span class="cite-bracket">]</span></a></sup> The forward index is <a href="Sorting_algorithm" title="Sorting algorithm">sorted</a> to transform it to an inverted index. The forward index is essentially a list of pairs consisting of a document and a word, collated by the document. Converting the forward index to an inverted index is only a matter of sorting the pairs by the words. In this regard, the inverted index is a word-sorted forward index.

</p>

<div class="mw-heading mw-heading3"><h3 id="Compression">Compression</h3></div>

<p>Generating or maintaining a large-scale search engine index represents a significant storage and processing challenge. Many search engines utilize a form of <a href="Data_compression" title="Data compression">compression</a> to reduce the size of the indices on <a href="Computer_storage" class="mw-redirect" title="Computer storage">disk</a>.<sup id="cite_ref-19" class="reference"><a href="#cite_note-19"><span class="cite-bracket">[</span>19<span class="cite-bracket">]</span></a></sup> Consider the following scenario for a full text, Internet search engine.

</p>

<ul><li>It takes 8 bits (or 1 <a href="Byte" title="Byte">byte</a>) to store a single character. Some <a href="Character_encoding" title="Character encoding">encodings</a> use 2 bytes per character<sup id="cite_ref-20" class="reference"><a href="#cite_note-20"><span class="cite-bracket">[</span>20<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-21" class="reference"><a href="#cite_note-21"><span class="cite-bracket">[</span>21<span class="cite-bracket">]</span></a></sup></li>

<li>The average number of characters in any given word on a page may be estimated at 5 (Wikipedia:Size comparisons)</li></ul>

<p>Given this scenario, an uncompressed index (assuming a non-<a href="Conflation" title="Conflation">conflated</a>, simple, index) for 2 billion web pages would need to store 500 billion word entries. At 1 byte per character, or 5 bytes per word, this would require 2500 gigabytes of storage space alone. This space requirement may be even larger for a fault-tolerant distributed storage architecture. Depending on the compression technique chosen, the index can be reduced to a fraction of this size. The tradeoff is the time and processing power required to perform compression and decompression.

</p><p>Notably, large scale search engine designs incorporate the cost of storage as well as the costs of electricity to power the storage. Thus compression is a measure of cost.

</p>

<div class="mw-heading mw-heading2"><h2 id="Document_parsing">Document parsing</h2></div>

<p>Document parsing breaks apart the components (words) of a document or other form of media for insertion into the forward and inverted indices. The words found are called <i>tokens</i>, and so, in the context of search engine indexing and <a href="Natural_language_processing" title="Natural language processing">natural language processing</a>, parsing is more commonly referred to as <a href="Tokenization_(lexical_analysis)" class="mw-redirect" title="Tokenization (lexical analysis)">tokenization</a>. It is also sometimes called word boundary disambiguation, <a href="Part-of-speech_tagging" title="Part-of-speech tagging">tagging</a>, <a href="Text_segmentation" title="Text segmentation">text segmentation</a>, <a href="Content_analysis" title="Content analysis">content analysis</a>, text analysis, <a href="Text_mining" title="Text mining">text mining</a>, <a href="Concordance_(publishing)" title="Concordance (publishing)">concordance</a> generation, <a href="Speech_segmentation" title="Speech segmentation">speech segmentation</a>, <a href="Lexical_analysis" title="Lexical analysis">lexing</a>, or <a href="Lexical_analysis" title="Lexical analysis">lexical analysis</a>. The terms 'indexing', 'parsing', and 'tokenization' are used interchangeably in corporate slang.

</p><p>Natural language processing is the subject of continuous research and technological improvement. Tokenization presents many challenges in extracting the necessary information from documents for indexing to support quality searching. Tokenization for indexing involves multiple technologies, the implementation of which are commonly kept as corporate secrets.

</p>

<div class="mw-heading mw-heading3"><h3 id="Challenges_in_natural_language_processing">Challenges in natural language processing</h3></div>

<dl><dt>Word boundary ambiguity</dt>

<dd>Native <a href="English_language" title="English language">English</a> speakers may at first consider tokenization to be a straightforward task, but this is not the case with designing a <a href="Multilingual" class="mw-redirect" title="Multilingual">multilingual</a> indexer. In digital form, the texts of other languages such as <a href="Chinese_language" title="Chinese language">Chinese</a> or <a href="Japanese_language" title="Japanese language">Japanese</a> represent a greater challenge, as words are not clearly delineated by <a href="Whitespace_(computer_science)" class="mw-redirect" title="Whitespace (computer science)">whitespace</a>. The goal during tokenization is to identify words for which users will search. Language-specific logic is employed to properly identify the boundaries of words, which is often the rationale for designing a parser for each language supported (or for groups of languages with similar boundary markers and syntax).</dd></dl>

<dl><dt>Language ambiguity</dt>

<dd>To assist with properly ranking matching documents, many search engines collect additional information about each word, such as its <a href="Language" title="Language">language</a> or <a href="Lexical_category" class="mw-redirect" title="Lexical category">lexical category</a> (<a href="Part_of_speech" title="Part of speech">part of speech</a>). These techniques are language-dependent, as the syntax varies among languages. Documents do not always clearly identify the language of the document or represent it accurately. In tokenizing the document, some search engines attempt to automatically identify the language of the document.</dd></dl>

<dl><dt>Diverse file formats</dt>

<dd>In order to correctly identify which bytes of a document represent characters, the file format must be correctly handled. Search engines that support multiple file formats must be able to correctly open and access the document and be able to tokenize the characters of the document.</dd></dl>

<dl><dt>Faulty storage</dt>

<dd>The quality of the natural language data may not always be perfect. An unspecified number of documents, particularly on the Internet, do not closely obey proper file protocol. <a href="Binary_data" title="Binary data">Binary</a> characters may be mistakenly encoded into various parts of a document. Without recognition of these characters and appropriate handling, the index quality or indexer performance could degrade.</dd></dl>

<div class="mw-heading mw-heading3"><h3 id="Tokenization">Tokenization</h3></div>

<p>Unlike <a href="Literacy" title="Literacy">literate</a> humans, computers do not understand the structure of a natural language document and cannot automatically recognize words and sentences. To a computer, a document is only a sequence of bytes. Computers do not 'know' that a space character separates words in a document. Instead, humans must program the computer to identify what constitutes an individual or distinct word referred to as a token. Such a program is commonly called a <a href="Tokenizer" class="mw-redirect" title="Tokenizer">tokenizer</a> or <a href="Parser" class="mw-redirect" title="Parser">parser</a> or <a href="Lexical_analysis" title="Lexical analysis">lexer</a>. Many search engines, as well as other natural language processing software, incorporate <a href="Comparison_of_parser_generators" title="Comparison of parser generators">specialized programs</a> for parsing, such as <a href="YACC" class="mw-redirect" title="YACC">YACC</a> or <a href="Lex_programming_tool" class="mw-redirect" title="Lex programming tool">Lex</a>.

</p><p>During tokenization, the parser identifies sequences of characters that represent words and other elements, such as punctuation, which are represented by numeric codes, some of which are non-printing control characters. The parser can also identify <a href="Entity_extraction" class="mw-redirect" title="Entity extraction">entities</a> such as <a href="Email" title="Email">email</a> addresses, phone numbers, and <a href="Uniform_Resource_Locator" class="mw-redirect" title="Uniform Resource Locator">URLs</a>. When identifying each token, several characteristics may be stored, such as the token's case (upper, lower, mixed, proper), language or encoding, lexical category (part of speech, like 'noun' or 'verb'), position, sentence number, sentence position, length, and line number.

</p>

<div class="mw-heading mw-heading3"><h3 id="Language_recognition">Language recognition</h3></div>

<p>If the search engine supports multiple languages, a common initial step during tokenization is to identify each document's language; many of the subsequent steps are language dependent (such as <a href="Stemming" title="Stemming">stemming</a> and <a href="Part_of_speech" title="Part of speech">part of speech</a> tagging). <a href="Language_identification" title="Language identification">Language recognition</a> is the process by which a computer program attempts to automatically identify, or categorize, the <a href="Language" title="Language">language</a> of a document. Other names for language recognition include language classification, language analysis, language identification, and language tagging. Automated language recognition is the subject of ongoing research in <a href="Natural_language_processing" title="Natural language processing">natural language processing</a>. Finding which language the words belongs to may involve the use of a language recognition chart.

</p>

<div class="mw-heading mw-heading3"><h3 id="Format_analysis">Format analysis</h3></div>

<p>If the search engine supports multiple <a href="File_format" title="File format">document formats</a>, documents must be prepared for tokenization. The challenge is that many document formats contain formatting information in addition to textual content. For example, <a href="HTML" title="HTML">HTML</a> documents contain HTML tags, which specify formatting information such as new line starts, bold emphasis, and <a href="Font" title="Font">font</a> size or <a href="Font_family" class="mw-redirect" title="Font family">style</a>. If the search engine were to ignore the difference between content and 'markup', extraneous information would be included in the index, leading to poor search results. Format analysis is the identification and handling of the formatting content embedded within documents which controls the way the document is rendered on a computer screen or interpreted by a software program. Format analysis is also referred to as structure analysis, format parsing, tag stripping, format stripping, text normalization, text cleaning and text preparation. The challenge of format analysis is further complicated by the intricacies of various file formats. Certain file formats are proprietary with very little information disclosed, while others are well documented. Common, well-documented file formats that many search engines support include:

</p>

<li><a href="ASCII" title="ASCII">ASCII</a> text files (a text document without specific computer readable formatting)</li>

<li><a href="Adobe_Systems" class="mw-redirect" title="Adobe Systems">Adobe</a>'s Portable Document Format (<a href="PDF" title="PDF">PDF</a>)</li>

<li><a href="PostScript" title="PostScript">PostScript</a> (PS)</li>

<li><a href="LaTeX" title="LaTeX">LaTeX</a></li>

<li><a href="UseNet" class="mw-redirect" title="UseNet">UseNet</a> netnews server formats</li>

<li><a href="XML" title="XML">XML</a> and derivatives like <a href="RSS" title="RSS">RSS</a></li>

<li><a href="Multimedia" title="Multimedia">Multimedia</a> <a href="Meta_data" class="mw-redirect" title="Meta data">meta data</a> formats like <a href="ID3" title="ID3">ID3</a></li>

<li><a href="Microsoft_Word" title="Microsoft Word">Microsoft Word</a></li>

<li><a href="Microsoft_Excel" title="Microsoft Excel">Microsoft Excel</a></li>

<li><a href="Microsoft_PowerPoint" title="Microsoft PowerPoint">Microsoft PowerPoint</a></li>

<li>IBM <a href="Lotus_Notes" class="mw-redirect" title="Lotus Notes">Lotus Notes</a></li></ul>

<p>Options for dealing with various formats include using a publicly available commercial parsing tool that is offered by the organization which developed, maintains, or owns the format, and writing a custom <a href="Parser" class="mw-redirect" title="Parser">parser</a>.

</p><p>Some search engines support inspection of files that are stored in a <a href="Compressor_(software)" title="Compressor (software)">compressed</a> or encrypted file format. When working with a compressed format, the indexer first decompresses the document; this step may result in one or more files, each of which must be indexed separately. Commonly supported <a href="List_of_archive_formats" title="List of archive formats">compressed file formats</a> include:

</p>

<ul><li><a href="ZIP_(file_format)" title="ZIP (file format)">ZIP</a> - Zip archive file</li>

<li><a href="RAR_(file_format)" title="RAR (file format)">RAR</a> - Roshal ARchive file</li>

<li><a href="Cabinet_(file_format)" title="Cabinet (file format)">CAB</a> - <a href="Microsoft_Windows" title="Microsoft Windows">Microsoft Windows</a> Cabinet File</li>

<li><a href="Gzip" title="Gzip">Gzip</a> - File compressed with gzip</li>

<li><a href="Bzip2" title="Bzip2">BZIP</a> - File compressed using bzip2</li>

<li><a href="Tar_(computing)" title="Tar (computing)">Tape ARchive (TAR)</a>, <a href="Unix" title="Unix">Unix</a> archive file, not (itself) compressed</li>

<li>TAR.Z, TAR.GZ or TAR.BZ2 - <a href="Unix" title="Unix">Unix</a> archive files compressed with Compress, GZIP or BZIP2</li></ul>

<p>Format analysis can involve quality improvement methods to avoid including 'bad information' in the index. Content can manipulate the formatting information to include additional content. Examples of abusing document formatting for <a href="Spamdexing" title="Spamdexing">spamdexing</a>:

</p>

<ul><li>Including hundreds or thousands of words in a section that is hidden from view on the computer screen, but visible to the indexer, by use of formatting (e.g. hidden <a href="Span_and_div" class="mw-redirect" title="Span and div">"div" tag</a> in <a href="HTML" title="HTML">HTML</a>, which may incorporate the use of <a href="CSS" title="CSS">CSS</a> or <a href="JavaScript" title="JavaScript">JavaScript</a> to do so).</li>

<li>Setting the foreground font color of words to the same as the background color, making words hidden on the computer screen to a person viewing the document, but not hidden to the indexer.</li></ul>

<div class="mw-heading mw-heading3"><h3 id="Section_recognition">Section recognition</h3></div>

<p>Some search engines incorporate section recognition, the identification of major parts of a document, prior to tokenization. Not all the documents in a corpus read like a well-written book, divided into organized chapters and pages. Many documents on the <a href="Internet" title="Internet">web</a>, such as newsletters and corporate reports, contain erroneous content and side-sections that do not contain primary material (that which the document is about). For example, articles on the Wikipedia website display a side menu with links to other web pages. Some file formats, like HTML or PDF, allow for content to be displayed in columns. Even though the content is displayed, or rendered, in different areas of the view, the raw markup content may store this information sequentially. Words that appear sequentially in the raw source content are indexed sequentially, even though these sentences and paragraphs are rendered in different parts of the computer screen. If search engines index this content as if it were normal content, the quality of the index and search quality may be degraded due to the mixed content and improper word proximity. Two primary problems are noted:

</p>

<ul><li>Content in different sections is treated as related in the index when in reality it is not</li>

<li>Organizational <i>side bar</i> content is included in the index, but the side bar content does not contribute to the meaning of the document, and the index is filled with a poor representation of its documents.</li></ul>

<p>Section analysis may require the search engine to implement the rendering logic of each document, essentially an abstract representation of the actual document, and then index the representation instead. For example, some content on the Internet is rendered via JavaScript. If the search engine does not render the page and evaluate the JavaScript within the page, it would not 'see' this content in the same way and would index the document incorrectly. Given that some search engines do not bother with rendering issues, many web page designers avoid displaying content via JavaScript or use the <a rel="nofollow" class="external text" href="https://worldwidenews.ru/2020/05/27/noscript-tag/">Noscript</a> tag to ensure that the web page is indexed properly. At the same time, this fact can also be <a href="Spamdexing" title="Spamdexing">exploited</a> to cause the search engine indexer to 'see' different content than the viewer.

</p>

<div class="mw-heading mw-heading3"><h3 id="HTML_priority_system">HTML priority system</h3></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .ambox{border:1px solid #a2a9b1;border-left:10px solid #36c;background-color:#fbfbfb;box-sizing:border-box}.mw-parser-output .ambox+link+.ambox,.mw-parser-output .ambox+link+style+.ambox,.mw-parser-output .ambox+link+link+.ambox,.mw-parser-output .ambox+.mw-empty-elt+link+.ambox,.mw-parser-output .ambox+.mw-empty-elt+link+style+.ambox,.mw-parser-output .ambox+.mw-empty-elt+link+link+.ambox{margin-top:-1px}html body.mediawiki .mw-parser-output .ambox.mbox-small-left{margin:4px 1em 4px 0;overflow:hidden;width:238px;border-collapse:collapse;font-size:88%;line-height:1.25em}.mw-parser-output .ambox-speedy{border-left:10px solid #b32424;background-color:#fee7e6}.mw-parser-output .ambox-delete{border-left:10px solid #b32424}.mw-parser-output .ambox-content{border-left:10px solid #f28500}.mw-parser-output .ambox-style{border-left:10px solid #fc3}.mw-parser-output .ambox-move{border-left:10px solid #9932cc}.mw-parser-output .ambox-protection{border-left:10px solid #a2a9b1}.mw-parser-output .ambox .mbox-text{border:none;padding:0.25em 0.5em;width:100%}.mw-parser-output .ambox .mbox-image{border:none;padding:2px 0 2px 0.5em;text-align:center}.mw-parser-output .ambox .mbox-imageright{border:none;padding:2px 0.5em 2px 0;text-align:center}.mw-parser-output .ambox .mbox-empty-cell{border:none;padding:0;width:1px}.mw-parser-output .ambox .mbox-image-div{width:52px}@media(min-width:720px){.mw-parser-output .ambox{margin:0 10%}}@media print{body.ns-0 .mw-parser-output .ambox{display:none!important}}

/* end https://en.wikipedia.org/ */

</style>

<p>Indexing often has to recognize the <a href="HTML" title="HTML">HTML</a> tags to organize priority. Indexing low priority to high margin to labels like <i>strong</i> and <i>link</i> to optimize the order of priority if those labels are at the beginning of the text could not prove to be relevant. Some indexers like <a href="Google" title="Google">Google</a> and <a href="Bing_(search_engine)" class="mw-redirect" title="Bing (search engine)">Bing</a> ensure that the <a href="Search_engine" title="Search engine">search engine</a> does not take the large texts as relevant source due to strong type system compatibility.<sup id="cite_ref-22" class="reference"><a href="#cite_note-22"><span class="cite-bracket">[</span>22<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Meta_tag_indexing">Meta tag indexing</h3></div>

<p>Meta tag indexing plays an important role in organizing and categorizing web content. Specific documents often contain embedded meta information such as author, keywords, description, and language. For HTML pages, the <a href="Meta_tag" class="mw-redirect" title="Meta tag">meta tag</a> contains keywords which are also included in the index. Earlier Internet <a href="Search_engine_technology" class="mw-redirect" title="Search engine technology">search engine technology</a> would only index the keywords in the meta tags for the forward index; the full document would not be parsed. At that time full-text indexing was not as well established, nor was <a href="Computer_hardware" title="Computer hardware">computer hardware</a> able to support such technology. The design of the HTML markup language initially included support for meta tags for the very purpose of being properly and easily indexed, without requiring tokenization.<sup id="cite_ref-23" class="reference"><a href="#cite_note-23"><span class="cite-bracket">[</span>23<span class="cite-bracket">]</span></a></sup>

</p><p>As the Internet grew through the 1990s, many <a href="Brick_and_mortar_business" class="mw-redirect" title="Brick and mortar business">brick-and-mortar corporations</a> went 'online' and established corporate websites. The keywords used to describe webpages (many of which were corporate-oriented webpages similar to product brochures) changed from descriptive to marketing-oriented keywords designed to drive sales by placing the webpage high in the search results for specific search queries. The fact that these keywords were subjectively specified was leading to <a href="Spamdexing" title="Spamdexing">spamdexing</a>, which drove many search engines to adopt full-text indexing technologies in the 1990s. Search engine designers and companies could only place so many 'marketing keywords' into the content of a webpage before draining it of all interesting and useful information. Given that conflict of interest with the business goal of designing user-oriented websites which were 'sticky', the <a href="Customer_lifetime_value" title="Customer lifetime value">customer lifetime value</a> equation was changed to incorporate more useful content into the website in hopes of retaining the visitor. In this sense, full-text indexing was more objective and increased the quality of search engine results, as it was one more step away from subjective control of search engine result placement, which in turn furthered research of full-text indexing technologies.

</p><p>In <a href="Desktop_search" title="Desktop search">desktop search</a>, many solutions incorporate meta tags to provide a way for authors to further customize how the search engine will index content from various files that is not evident from the file content. Desktop search is more under the control of the user, while Internet search engines must focus more on the full text index.{{cn|date=August 2025}

</p>

<ul><li><a href="Controlled_vocabulary" title="Controlled vocabulary">Controlled vocabulary</a></li>

<li><a href="Database_index" title="Database index">Database index</a></li>

<li><a href="Full-text_search" title="Full-text search">Full-text search</a></li>

<li><a href="Information_extraction" title="Information extraction">Information extraction</a></li>

<li><a href="Key_Word_in_Context" title="Key Word in Context">Key Word in Context</a></li>

<li><a href="Selection-based_search" title="Selection-based search">Selection-based search</a></li>

<li><a href="Text_retrieval" class="mw-redirect" title="Text retrieval">Text retrieval</a></li>

<li><a href="Information_literacy" title="Information literacy">Information literacy</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

<li id="cite_note-1"><span class="mw-cite-backlink"><b><a href="#cite_ref-1">^</a></b></span> <span class="reference-text">Clarke, C., Cormack, G.: Dynamic Inverted Indexes for a Distributed Full-Text Retrieval System. TechRep MT-95-01, University of Waterloo, February 1995.</span>

</li>

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://web.archive.org/web/20060512074748/http://www.ee.columbia.edu:80/~dpwe/papers/Wang03-shazam.pdf">"An Industrial-Strength Audio Search Algorithm"</a> <span class="cs1-format">(PDF)</span>. Archived from <a rel="nofollow" class="external text" href="http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf">the original</a> <span class="cs1-format">(PDF)</span> on 2006-05-12.</cite></span>

</li>

<li id="cite_note-3"><span class="mw-cite-backlink"><b><a href="#cite_ref-3">^</a></b></span> <span class="reference-text">Charles E. Jacobs, Adam Finkelstein, David H. Salesin. <a rel="nofollow" class="external text" href="http://grail.cs.washington.edu/projects/query/mrquery.pdf">Fast Multiresolution Image Querying</a>. Department of Computer Science and Engineering, University of Washington. 1995. Verified Dec 2006</span>

</li>

<li id="cite_note-4"><span class="mw-cite-backlink"><b><a href="#cite_ref-4">^</a></b></span> <span class="reference-text">Brown, E.W.: Execution Performance Issues in Full-Text Information Retrieval. Computer Science Department, University of Massachusetts Amherst, Technical Report 95-81, October 1995.</span>

</li>

<li id="cite_note-5"><span class="mw-cite-backlink"><b><a href="#cite_ref-5">^</a></b></span> <span class="reference-text">Cutting, D., Pedersen, J.: Optimizations for dynamic inverted index maintenance. Proceedings of SIGIR, 405-411, 1990.</span>

</li>

<li id="cite_note-6"><span class="mw-cite-backlink"><b><a href="#cite_ref-6">^</a></b></span> <span class="reference-text"><a rel="nofollow" class="external text" href="http://dev.mysql.com/doc/refman/5.1/en/partitioning-linear-hash.html">Linear Hash Partitioning</a>. MySQL 5.1 Reference Manual. Verified Dec 2006</span>

</li>

<li id="cite_note-7"><span class="mw-cite-backlink"><b><a href="#cite_ref-7">^</a></b></span> <span class="reference-text"><a rel="nofollow" class="external text" href="https://xlinux.nist.gov/dads/HTML/trie.html">trie</a>, <a rel="nofollow" class="external text" href="https://www.nist.gov/dads">Dictionary of Algorithms and Data Structures</a>, <a rel="nofollow" class="external text" href="http://www.nist.gov">U.S. National Institute of Standards and Technology</a>.</span>

</li>

<li id="cite_note-Gus97-8"><span class="mw-cite-backlink"><b><a href="#cite_ref-Gus97_8-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFGusfield1999" class="citation book cs1">Gusfield, Dan (1999) [1997]. <i>Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology</i>. US: Cambridge University Press. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-521-58519-8</bdi>.</cite>.</span>

</li>

<li id="cite_note-9"><span class="mw-cite-backlink"><b><a href="#cite_ref-9">^</a></b></span> <span class="reference-text">Black, Paul E., <a rel="nofollow" class="external text" href="https://xlinux.nist.gov/dads/HTML/invertedIndex.html">inverted index</a>, <a rel="nofollow" class="external text" href="https://www.nist.gov/dads">Dictionary of Algorithms and Data Structures</a>, <a rel="nofollow" class="external text" href="http://www.nist.gov">U.S. National Institute of Standards and Technology</a> Oct 2006. Verified Dec 2006.</span>

</li>

<li id="cite_note-10"><span class="mw-cite-backlink"><b><a href="#cite_ref-10">^</a></b></span> <span class="reference-text">C. C. Foster, Information retrieval: information storage and retrieval using AVL trees, Proceedings of the 1965 20th national conference, p.192-205, August 24–26, 1965, Cleveland, Ohio, United States</span>

</li>

<li id="cite_note-11"><span class="mw-cite-backlink"><b><a href="#cite_ref-11">^</a></b></span> <span class="reference-text">Landauer, W. I.: The balanced tree and its utilization in information retrieval. IEEE Trans. on Electronic Computers, Vol. EC-12, No. 6, December 1963.</span>

</li>

<li id="cite_note-12"><span class="mw-cite-backlink"><b><a href="#cite_ref-12">^</a></b></span> <span class="reference-text"><a rel="nofollow" class="external text" href="http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2006T13">Google Ngram Datasets</a> <a rel="nofollow" class="external text" href="https://web.archive.org/web/20130929081544/http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2006T13">Archived</a> 2013-09-29 at the <a href="Wayback_Machine" title="Wayback Machine">Wayback Machine</a> for sale at <a rel="nofollow" class="external text" href="http://www.ldc.upenn.edu/">LDC</a> Catalog</span>

</li>

<li id="cite_note-13"><span class="mw-cite-backlink"><b><a href="#cite_ref-13">^</a></b></span> <span class="reference-text">Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Google, Inc. OSDI. 2004.</span>

</li>

<li id="cite_note-14"><span class="mw-cite-backlink"><b><a href="#cite_ref-14">^</a></b></span> <span class="reference-text">Grossman, Frieder, Goharian. <a rel="nofollow" class="external text" href="http://www.cs.clemson.edu/~juan/CPSC862/Concept-50/IR-Basics-of-Inverted-Index.pdf">IR Basics of Inverted Index</a>. 2002. Verified Aug 2011.</span>

</li>

<li id="cite_note-15"><span class="mw-cite-backlink"><b><a href="#cite_ref-15">^</a></b></span> <span class="reference-text">Tang, Hunqiang. <a href="Sandhya_Dwarkadas" title="Sandhya Dwarkadas">Dwarkadas, Sandhya</a>. "Hybrid Global Local Indexing for Efficient

Peer to Peer Information Retrieval". University of Rochester. Pg 1. <a rel="nofollow" class="external free" href="http://www.cs.rochester.edu/u/sandhya/papers/nsdi04.ps">http://www.cs.rochester.edu/u/sandhya/papers/nsdi04.ps</a></span>

</li>

<li id="cite_note-16"><span class="mw-cite-backlink"><b><a href="#cite_ref-16">^</a></b></span> <span class="reference-text"><cite id="CITEREFBüttcherClarkeCormack2016" class="citation book cs1">Büttcher, Stefan; Clarke, Charles L. A.; Cormack, Gordon V. (2016). <i>Information retrieval: implementing and evaluating search engines</i> (First MIT Press paperback ed.). Cambridge, Massachusetts London, England: The MIT Press. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-262-52887-0</bdi>.</cite></span>

</li>

<li id="cite_note-17"><span class="mw-cite-backlink"><b><a href="#cite_ref-17">^</a></b></span> <span class="reference-text">Tomasic, A., et al.: Incremental Updates of Inverted Lists for Text Document Retrieval. Short Version of Stanford University Computer Science Technical Note STAN-CS-TN-93-1, December, 1993.</span>

</li>

<li id="cite_note-18"><span class="mw-cite-backlink"><b><a href="#cite_ref-18">^</a></b></span> <span class="reference-text">Sergey Brin and Lawrence Page. <a rel="nofollow" class="external text" href="http://infolab.stanford.edu/~backrub/google.html">The Anatomy of a Large-Scale Hypertextual Web Search Engine</a>. <a href="Stanford_University" title="Stanford University">Stanford University</a>. 1998. Verified Dec 2006.</span>

</li>

<li id="cite_note-19"><span class="mw-cite-backlink"><b><a href="#cite_ref-19">^</a></b></span> <span class="reference-text">H.S. Heaps. Storage analysis of a compression coding for a document database. 1NFOR, I0(i):47-61, February 1972.</span>

</li>

<li id="cite_note-20"><span class="mw-cite-backlink"><b><a href="#cite_ref-20">^</a></b></span> <span class="reference-text"><a rel="nofollow" class="external text" href="https://www.unicode.org/faq/basic_q.html#15">The Unicode Standard - Frequently Asked Questions</a>. Verified Dec 2006.</span>

</li>

<li id="cite_note-21"><span class="mw-cite-backlink"><b><a href="#cite_ref-21">^</a></b></span> <span class="reference-text"><a rel="nofollow" class="external text" href="https://web.archive.org/web/20010209140313/http://www.uplink.freeuk.com/data.html">Storage estimates</a>. Verified Dec 2006.</span>

</li>

<li id="cite_note-22"><span class="mw-cite-backlink"><b><a href="#cite_ref-22">^</a></b></span> <span class="reference-text">Google Webmaster Tools, "Hypertext Markup Language 5", Conference for SEO January 2012.</span>

</li>

<li id="cite_note-23"><span class="mw-cite-backlink"><b><a href="#cite_ref-23">^</a></b></span> <span class="reference-text">Berners-Lee, T., "Hypertext Markup Language - 2.0", RFC 1866, Network Working Group, November 1995.</span>

</li>

</ol></div></div>

<div class="mw-heading mw-heading2"><h2 id="Further_reading">Further reading</h2></div>

<ul><li>R. Bayer and E. McCreight. Organization and maintenance of large ordered indices. Acta Informatica, 173-189, 1972.</li>

<li><a href="Donald_E._Knuth" class="mw-redirect" title="Donald E. Knuth">Donald E. Knuth</a>. <a href="The_Art_of_Computer_Programming" title="The Art of Computer Programming">The Art of Computer Programming</a>, volume 1 (3rd ed.): fundamental algorithms, Addison Wesley Longman Publishing Co. Redwood City, CA, 1997.</li>

<li><a href="Donald_E._Knuth" class="mw-redirect" title="Donald E. Knuth">Donald E. Knuth</a>. The art of computer programming, volume 3: (2nd ed.) sorting and searching, Addison Wesley Longman Publishing Co. Redwood City, CA, 1998.</li>

<li><a href="Gerald_Salton" class="mw-redirect" title="Gerald Salton">Gerald Salton</a>. Automatic text processing, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1988.</li>

<li><a href="Gerard_Salton" title="Gerard Salton">Gerard Salton</a>. Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986.</li>

<li><a href="Gerard_Salton" title="Gerard Salton">Gerard Salton</a>. Lesk, M.E.: Computer evaluation of indexing and text processing. Journal of the ACM. January 1968.</li>

<li><a href="Gerard_Salton" title="Gerard Salton">Gerard Salton</a>. The SMART Retrieval System - Experiments in Automatic Document Processing. Prentice Hall Inc., Englewood Cliffs, 1971.</li>

<li><a href="Gerard_Salton" title="Gerard Salton">Gerard Salton</a>. The Transformation, Analysis, and Retrieval of Information by Computer, Addison-Wesley, Reading, Mass., 1989.</li>

<li>Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Chapter 8. ACM Press 1999.</li>

<li>G. K. Zipf. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949.</li>

<li>Adelson-Velskii, G.M., Landis, E. M.: An information organization algorithm. DANSSSR, 146, 263-266 (1962).</li>

<li><a href="Edward_H._Sussenguth_Jr." class="mw-redirect" title="Edward H. Sussenguth Jr.">Edward H. Sussenguth Jr.</a>, Use of tree structures for processing files, Communications of the ACM, v.6 n.5, p. 272-279, May 1963</li>

<li>Harman, D.K., et al.: Inverted files. In Information Retrieval: Data Structures and Algorithms, Prentice-Hall, pp 28–43, 1992.</li>

<li>Lim, L., et al.: Characterizing Web Document Change, LNCS 2118, 133–146, 2001.</li>

<li>Lim, L., et al.: Dynamic Maintenance of Web Indexes Using Landmarks. Proc. of the 12th W3 Conference, 2003.</li>

<li>Moffat, A., Zobel, J.: Self-Indexing Inverted Files for Fast Text Retrieval. ACM TIS, 349–379, October 1996, Volume 14, Number 4.</li>

<li><a href="Kurt_Mehlhorn" title="Kurt Mehlhorn">Mehlhorn, K.</a>: Data Structures and Efficient Algorithms, Springer Verlag, EATCS Monographs, 1984.</li>

<li><a href="Kurt_Mehlhorn" title="Kurt Mehlhorn">Mehlhorn, K.</a>, <a href="Mark_Overmars" title="Mark Overmars">Overmars, M.H.</a>: Optimal Dynamization of Decomposable Searching Problems. IPL 12, 93–98, 1981.</li>

<li><a href="Kurt_Mehlhorn" title="Kurt Mehlhorn">Mehlhorn, K.</a>: Lower Bounds on the Efficiency of Transforming Static Data Structures into Dynamic Data Structures. Math. Systems Theory 15, 1–16, 1981.</li>

<li>Koster, M.: ALIWEB: Archie-Like indexing in the Web. Computer Networks and ISDN Systems, Vol. 27, No. 2 (1994) 175-182 (also see Proc. First Int'l World Wide Web Conf., Elsevier Science, Amsterdam, 1994, pp. 175–182)</li>

<li><a href="Serge_Abiteboul" title="Serge Abiteboul">Serge Abiteboul</a> and <a href="Victor_Vianu" title="Victor Vianu">Victor Vianu</a>. <a rel="nofollow" class="external text" href="http://dbpubs.stanford.edu:8090/pub/showDoc.Fulltext?lang=en&doc=1996-20&format=text&compression=&name=1996-20.text">Queries and Computation on the Web</a>. Proceedings of the International Conference on Database Theory. Delphi, Greece 1997.</li>

<li>Ian H Witten, Alistair Moffat, and Timothy C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. New York: Van Nostrand Reinhold, 1994.</li>

<li>A. Emtage and P. Deutsch, "Archie--An Electronic Directory Service for the Internet." Proc. Usenix Winter 1992 Tech. Conf., Usenix Assoc., Berkeley, Calif., 1992, pp. 93–110.</li>

<li>M. Gray, <a rel="nofollow" class="external text" href="https://www.mit.edu/people/mkgray/net/">World Wide Web Wanderer</a>.</li>

<li>D. Cutting and J. Pedersen. "Optimizations for Dynamic Inverted Index Maintenance." Proceedings of the 13th International Conference on Research and Development in Information Retrieval, pp. 405–411, September 1990.</li>

<li>Stefan Büttcher, Charles L. A. Clarke, and Gordon V. Cormack. <a rel="nofollow" class="external text" href="http://www.ir.uwaterloo.ca/book/">Information Retrieval: Implementing and Evaluating Search Engines</a> <a rel="nofollow" class="external text" href="https://web.archive.org/web/20201005195805/http://www.ir.uwaterloo.ca/book/">Archived</a> 2020-10-05 at the <a href="Wayback_Machine" title="Wayback Machine">Wayback Machine</a>. MIT Press, Cambridge, Mass., 2010.</li></ul>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1236075235">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbox{box-sizing:border-box;border:1px solid #a2a9b1;width:100%;clear:both;font-size:88%;text-align:center;padding:1px;margin:1em auto 0}.mw-parser-output .navbox .navbox{margin-top:0}.mw-parser-output .navbox+.navbox,.mw-parser-output .navbox+.navbox-styles+.navbox{margin-top:-1px}.mw-parser-output .navbox-inner,.mw-parser-output .navbox-subgroup{width:100%}.mw-parser-output .navbox-group,.mw-parser-output .navbox-title,.mw-parser-output .navbox-abovebelow{padding:0.25em 1em;line-height:1.5em;text-align:center}.mw-parser-output .navbox-group{white-space:nowrap;text-align:right}.mw-parser-output .navbox,.mw-parser-output .navbox-subgroup{background-color:#fdfdfd}.mw-parser-output .navbox-list{line-height:1.5em;border-color:#fdfdfd}.mw-parser-output .navbox-list-with-group{text-align:left;border-left-width:2px;border-left-style:solid}.mw-parser-output tr+tr>.navbox-abovebelow,.mw-parser-output tr+tr>.navbox-group,.mw-parser-output tr+tr>.navbox-image,.mw-parser-output tr+tr>.navbox-list{border-top:2px solid #fdfdfd}.mw-parser-output .navbox-title{background-color:#ccf}.mw-parser-output .navbox-abovebelow,.mw-parser-output .navbox-group,.mw-parser-output .navbox-subgroup .navbox-title{background-color:#ddf}.mw-parser-output .navbox-subgroup .navbox-group,.mw-parser-output .navbox-subgroup .navbox-abovebelow{background-color:#e6e6ff}.mw-parser-output .navbox-even{background-color:#f7f7f7}.mw-parser-output .navbox-odd{background-color:transparent}.mw-parser-output .navbox .hlist td dl,.mw-parser-output .navbox .hlist td ol,.mw-parser-output .navbox .hlist td ul,.mw-parser-output .navbox td.hlist dl,.mw-parser-output .navbox td.hlist ol,.mw-parser-output .navbox td.hlist ul{padding:0.125em 0}.mw-parser-output .navbox .navbar{display:block;font-size:100%}.mw-parser-output .navbox-title .navbar{float:left;text-align:left;margin-right:0.5em}body.skin--responsive .mw-parser-output .navbox-image img{max-width:none!important}@media print{body.ns-0 .mw-parser-output .navbox{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></div><div role="navigation" class="navbox" aria-labelledby="Internet_search65" style="padding:3px"><table class="nowraplinks mw-collapsible autocollapse navbox-inner" style="border-spacing:0;background:transparent;color:inherit"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div id="Internet_search65" style="font-size:114%;margin:0 4em"><a href="Internet_search" class="mw-redirect" title="Internet search">Internet search</a></div></th></tr><tr><th scope="row" class="navbox-group" style="width:1%">Types</th><td class="navbox-list-with-group navbox-list navbox-odd hlist" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Web_search_engine" class="mw-redirect" title="Web search engine">Web search engine</a> (<a href="List_of_search_engines" title="List of search engines">List</a>)</li>

<li><a href="Metasearch_engine" title="Metasearch engine">Metasearch engine</a></li>

<li><a href="Multimedia_search" title="Multimedia search">Multimedia search</a></li>

<li><a href="Collaborative_search_engine" title="Collaborative search engine">Collaborative search engine</a></li>

<li><a href="Cross-language_search" class="mw-redirect" title="Cross-language search">Cross-language search</a></li>

<li><a href="Local_search_(Internet)" title="Local search (Internet)">Local search</a></li>

<li><a href="Vertical_search" title="Vertical search">Vertical search</a></li>

<li><a href="Social_search" title="Social search">Social search</a></li>

<li><a href="Image_retrieval" title="Image retrieval">Image search</a></li>

<li><a href="Audio_search_engine" title="Audio search engine">Audio search</a></li>

<li><a href="Video_search_engine" title="Video search engine">Video search engine</a></li>

<li><a href="Enterprise_search" title="Enterprise search">Enterprise search</a></li>

<li><a href="Semantic_search" title="Semantic search">Semantic search</a></li>

<li><a href="Natural_language_search_engine" class="mw-redirect" title="Natural language search engine">Natural language search engine</a></li>

<li><a href="Voice_search" title="Voice search">Voice search</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Tools</th><td class="navbox-list-with-group navbox-list navbox-even hlist" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Cross-language_information_retrieval" title="Cross-language information retrieval">Cross-language information retrieval</a></li>

<li><a href="Search_by_sound" title="Search by sound">Search by sound</a></li>

<li><a href="Search_engine_marketing" title="Search engine marketing">Search engine marketing</a></li>

<li><a href="Search_engine_optimization" title="Search engine optimization">Search engine optimization</a></li>

<li><a href="Evaluation_measures_(information_retrieval)" title="Evaluation measures (information retrieval)">Evaluation measures</a></li>

<li>Search oriented architecture</li>

<li><a href="Selection-based_search" title="Selection-based search">Selection-based search</a></li>

<li><a href="Document_retrieval" title="Document retrieval">Document retrieval</a></li>

<li><a href="Text_mining" title="Text mining">Text mining</a></li>

<li><a href="Web_crawler" title="Web crawler">Web crawler</a></li>

<li><a href="Multisearch" title="Multisearch">Multisearch</a></li>

<li><a href="Federated_search" title="Federated search">Federated search</a></li>

<li><a href="Search_aggregator" title="Search aggregator">Search aggregator</a></li>

<li><a href="Index_(search_engine)" class="mw-redirect" title="Index (search engine)">Index</a>/<a href="Web_indexing" title="Web indexing">Web indexing</a></li>

<li><a href="Focused_crawler" title="Focused crawler">Focused crawler</a></li>

<li><a href="Spider_trap" title="Spider trap">Spider trap</a></li>

<li><a href="Robots_exclusion_standard" class="mw-redirect" title="Robots exclusion standard">Robots exclusion standard</a></li>

<li><a href="Distributed_web_crawling" title="Distributed web crawling">Distributed web crawling</a></li>

<li><a href="Web_archiving" title="Web archiving">Web archiving</a></li>

<li><a href="Website_mirroring_software" class="mw-redirect" title="Website mirroring software">Website mirroring software</a></li>

<li><a href="Web_query" title="Web query">Web query</a></li>

<li><a href="Web_query_classification" title="Web query classification">Web query classification</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Protocols<br>and standards</th><td class="navbox-list-with-group navbox-list navbox-odd hlist" style="width:100%;padding:0"><div style="padding:0 0.25em">

<li><a href="Search/Retrieve_Web_Service" title="Search/Retrieve Web Service">Search/Retrieve Web Service</a></li>

<li><a href="Search/Retrieve_via_URL" title="Search/Retrieve via URL">Search/Retrieve via URL</a></li>

<li><a href="OpenSearch_(specification)" title="OpenSearch (specification)">OpenSearch</a></li>

<li><a href="Representational_State_Transfer" class="mw-redirect" title="Representational State Transfer">Representational State Transfer</a></li>

<li><a href="Wide_area_information_server" title="Wide area information server">Wide area information server</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">See also</th><td class="navbox-list-with-group navbox-list navbox-even hlist" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Search_engine_(computing)" title="Search engine (computing)">Search engine</a></li>

<li><a href="Desktop_search" title="Desktop search">Desktop search</a></li>

<li><a href="Online_search" title="Online search">Online search</a></li></ul>

</div></td></tr></tbody></table></div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-08-04" href="https://en.wikipedia.org/wiki/?title=Search_engine_indexing&oldid=1304209568">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>